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1 Introduction 

Software Engineering is a knowledge intensive activ- 
ity that involves defining, designing, developing, and 
maintaining software systems. In order to build ef- 
fective systems to support Software Engeneering ac- 
tivities, Artificial Intelligence techniques are needed. 
The application of Artificial Intelligence technology 
to Software Engineering is called Knowledge-based 
Software Engineering (KBSE) [Lowry & Duran, 1989]. 
The goal of KBSE is to change the software life cycle 
such that software maintenance and evolution occur by 
modifying the specifications and then rederiving the 
implementation rather than by directly modifying the 
implementation. The use of domain knowledge in de- 
veloping KBSE systems is crucial. 

Our work is mainly related to one area of 
KBSE that is called automatic specification acqui- 
sition. One example is the WATSON prototype 
[Kelly h Nonnenmann, 1991] on which our current 
work is based. WATSON is an automatic program- 
ming system for formalizing specifications for tele- 
phone switching software mainly restricted to POTS, 
i.e. t plain old telephone service. 

Other examples of such systems are IDeA 
and Ozym. The Intelligent Design Aid (IDeA) 
[Lubars it Harandi, 1987] performs knowledge- based 
refinement of specifications and design. IDeA gives 
incremental feedback on completeness and consis- 
tency using domain-specific abstract design schemas. 
The idea behind Ozym [Iscoe et ai, 1989] is to spec- 
ify and implement applications programs for non- 
programmers and non-domain-experts by modeling do- 
main knowledge. 

However, despite two decades of moderately suc- 
cessful research, there have been few practical demon- 
strations of the utility of Artificial Intelligence tech- 
niques to support Software Engineering activities 
[Barstow, 1987] other than such prototypes as men- 
tioned above. Our current approach differentiates it- 
self from these other approaches in two antagonistic 
ways: On the one hand, we address a large and com- 
plex real-world problem instead of a “toy domain” as 
in many research prototypes. On the other hand, to 


allow such scaling, we had to relax the ambitious goal 
of complete automatic programming , to the easier task 
of automatic testing. 

2 KITSS Overview 

In the Knowledge-Based Interactive Test Script Sys- 
tem (KITSS), we have taken this philosophy and ap- 
plied it to the task of functional software testing. In 
functional testing, the internal design and structure of 
the program are ignored. It corresponds directly to 
uncovering discrepancies in the program’s behavior as 
viewed from the outside world. This type of testing 
has been called black box testing because, like a black 
box in hardware, one is only interested in how the in- 
put relates to the output. The resulting tests are then 
executed in a simulated customer environment which 
corresponds to verifying that the system fulfills its in- 
tended purpose. 

Tests are by definition correct but not exhaustive. 
KITSS checks and augments given tests and generates 
related new ones but does not generate the full spec- 
ification as in WATSON. KITSS can be seen as per- 
forming testing from examples. KITSS’ strength lies 
in its very domain-specific approach [Barstow, 1985] 
and customized reasoning procedures. It will change 
the software life cycle by modifying the functional 
tests and then rederiving the system tests which cor- 
responds to finding and eliminating software prob- 
lems early in the development process as in the KBSE 
paradigm. Therefore, we designed KITSS to be well 
integrated into our existing design and development 
process [Nonnenmann As Eddy, 1991]. 

KITSS achieves this integration by using the same 
expressive and unobtrusive input medium, namely test 
cases . They describe in English the high-level details 
of the external design and are written before coding 
begins. KITSS also produces the same output as be- 
fore, executable test scripts written in an in-house test 
automation language. These are low-level descriptions 
derived from test cases for specific test equipment. 

To support this integration, KITSS has a natural 
language processor that is trained in the domain’s tech- 
nical dialect [Jones Eisner, 1992] and converts the 
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Figure 1: KITSS Architecture 


test cases into a formal representation that is au- 
dited for coverage and sanity. To accomplish this, 
KITSS uses a customized theorem prover-based ana- 
lyzer (based on WATSON technology) and a hybrid 
knowledge base as the domain model using both a 
static terminological logic and a dynamic temporal 
logic. These two modules have been feasible only 
due to the domain-specific knowledge-based approach 
taken in KITSS. Finally, a translator takes the cor- 
rected test case and converts it from temporal logic 
into a test script language that can exercise the switch 
using dedicated test equipment. Figure 1 shows the 
overall architecture of KITSS. 

In summary, KITSS helps the test process by gen- 
erating more tests of better quality and by allow- 
ing more frequent regression testing through automa- 
tion. Furthermore, tests are generated earlier, i.e., 
during the development phase not after , which should 
detect problems earlier, thus resulting in reduced 
maintenance costs (for more details on KITSS see 
[Nonnenmann <& Eddy, 1992]). 

3 Knowledge Representation Issues 

As we used a highly domain-specific approach, the do- 
main model is one of the center pieces of KITSS. In the 
following section we will highlight the key design deci- 
sions made and the knowledge representations chosen. 

Testing is a very knowledge intensive task. It in- 
volves experience with the switch hardware and testing 
equipment as well as an understanding of the switch 
software with its several hundred features and many 
more interactions. There are many binders of feature 
descriptions for PBX software, but no concise formal- 
izations of the domain were available before KITSS. 
The focus of KITSS and the domain model is on an 
end-user’s point of view, i.e., on (physical and soft- 
ware) objects that the user can manipulate. Figure 2 


gives an overview of KITSS’ domain model. 

The static model represents all telephony objects, 
data, and conditions that do not have a tempo- 
ral extent but may have states or histories. It de- 
scribes major hardware components, processes, log- 
ical resources, the current test setup, the dial plan 
and the current feature assignments. All static parts 
of the domain model are implemented in CLASSIC 
[Brachman et ai, 1990], which belongs to the class of 
terminological logics (e.g. KL-ONE). 

The dynamic model defines the dynamic aspects of 
the switch behavior. These are constraints that have to 
be fulfilled during testing as well as the predicates they 
are defined upon. Objects include predicates, stimuli 
which can be either primitive or abstract, and observ- 
ables. Additionally, the dynamic model includes in- 
variants and rules as integrity constraints. Invariants 
are assertions which describe only a single state, but 
are true in all states. These are among the most im- 
portant pieces of domain knowledge as they describe 
basic telephony behavior as well as the look & feel of 
the switch. Rules describe low-level behavior in tele- 
phony. This is mostly signaling behavior. 

Representing the dynamic model we required ex- 
pressive power beyond CLASSIC or terminological log- 
ics, which are not well-suited for representing plan-like 
knowledge. We therefore used the WATSON Theo- 
rem Prover, a linear-time first-order resolution theorem 
prover with a weak temporal logic. This non-standard 
logic has five modal operators holds, occurs , issues, be- 
gins, and ends. As an action occurs , the response to 
that action may endure until some other action occurs 
or it may be transient. An enduring actions begins and 
holds until, in response to some other action, it ends. 
In the transient case, the switch merely issues the re- 
sponse. These modals are sufficient to represent all 
temporal aspects of our domain. The theorem proving 
is only tractable due to the tight integration between 
knowledge representation and reasoning. 

In adding the dynamic model, we were able to in- 
crease the expressive power of our domain model and to 
increase the reasoning capabilities as well. The integra- 
tion of the hybrid pieces did produce some problems, 
for example, deciding which components belonged in 
which piece. However, this decision was facilitated be- 
cause of our design choice to represent all dynamic as- 
pects of the system in our temporal logic and to keep 
everything else in CLASSIC. 

The domain model consists of over 600 domain con- 
cepts, over 1,700 domain individuals, and more than 
160 temporal axioms. 

The domain model was built in the initial phase of 
the project as the reasoning modules depended on the 
underlying representations being created first. In this 
phase the domain model changed constantly as we still 
enhanced our understanding of the domain. Then, we 
left the domain model mainly unchanged through the 
development phases until a milestone was reached. We 
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Figure 2: KITSS Domain Model 


then typically performed a major revision of the do- 
main model based on problems encountered- The do- 
main model went through three iterations like that and 
has been stable since. Of course, we continually add 
new knowledge but the representations are mainly un- 
changed. 

Although we anticipate that the domain model will 
grow only linear with the number of features cov- 
ered, we already had great difficulty in acquiring new 
knowledge and maintaining the existing domain model 
as both tasks have been done completely manual so 
far. Therefore, knowledge acquisition and maintenance 
support is crucial. At least we have gained an under- 
standing on how to design such automated tools. The 
above experience is the main motivation for another 
approach included as a proposal in this workshop’s pro- 
ceedings [Hall, 1992]. 

4 Status 

At last year’s ASD workshop, we initially reported on 
KITSS. Since then, we had a major evaluation of the 
KITSS prototype with the following results. 

System execution speed has not been a bottleneck 
due to continued specialization of the inference capa- 
bility. However, it is not clear how long such optimiza- 
tions can avoid potential intractability of the theorem 
prover. Another result we found was that it is easier 
to train the natural language processor than it is to 
achieve coherence in the reasoning audits. The initial 
scaling phase from the proof-of-principle demo (3 test 
cases) to the first prototype (38 test cases) was suc- 
cessful but took too long and KITSS was too brittle in 
general. Although KITSS has been designed from the 
beginning as an interactive system (the “I” in KITSS), 
we pushed the machine initiative in this scaling phase 
as far as possible as an experiment. However, this was 
the limiting factor for rapid progress. 


The current schedule is to expand KITSS to cover a 
few hundred test cases in the next couple of months. 
To achieve such scaling, we changed some of KITSS’ 
design to make it much more robust. Despite the fact 
that the natural language processor performed well, we 
augmented it with a paraphrasing mechanism, i. e. , in 
case the English input cannot be understood, the user 
can rephrase this input using a paraphrase language 
based on our temporal logic. When the analyzer en- 
counters problems, it intensely questions the user to 
explain unclear passages of test cases. This approach 
has only been possible because KITSS is very respon- 
sive. Additionally, we changed all reasoning modules 
to produce “soft- failu res” , i.e, y in case the system fails 
to fully understand the test case it still continues to 
translate user inputs as an assistant. 

In general, our strategy has shifted from a nearly 
fully automatic system to one that is much more in- 
teractive and might resort to the assistant paradigm 
occasionally. However, we still have the full KITSS 
approach in place so that we can improve e.g. the an- 
alyzer incrementally without any change for the user 
other than reduced interactions. 

5 From Research Into Practice 

In 1987, the initial prototype of WATSON was com- 
pleted. Today, five years later, KITSS is a prototype 
with a more restricted scope in the same domain al- 
though with broader coverage. So where is the big 
progress? Or in other words: How do we get from a 
research prototype to a deployed real-world system? 
Is this “just” technology transfer? Another question 
might also be: Can we build a research prototype in 
a toy domain and expect it to work on a real-world 
problem? 

The answers to these questions 1 are not easy, but we 
would like to give at least partial answers based on our 
experience. 

In scaling from WATSON’s toy domain to KITSS’ 
real-world telephony domain we had to address a num- 
ber of new research issues (which we can just list here 
without further explanation). For example, we had 
to extend our temporal logic, restructure the domain 
model into a hybrid one (static/dynamic), create tele- 
phony “micromodels” and understand their interac- 
tions, reason with multiple agents instead of single 
ones, significantly enhance the planning component, 
understand the “purpose” of inputs to be able to gener- 
alize and specialize them, perform more complex non- 
monotonic reasoning etc. Additionally, we had to in- 
corporate and customize a natural language module to 
cover input test cases in English instead of a limited 
scenario language. 

KITSS being so different in all these respects, WAT- 
SON cannot be seen as a core system to which we just 

1 These questions are based on a persona] discussion with 
Ron Brachman. 
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added domain knowledge. For KITSS, we had to ba- 
sically rewrite and enhance WATSON and add com- 
pletely new modules. We see KITSS based on technol- 
ogy that WATSON has proven feasible. Therefore, we 
do not see KITSS as technology transfer at all but as 
a research project that covers a real-world domain. 

Yet despite such research progress, we were still 
required to further change KITSS’ design to achieve 
practical solutions (see Section 4). No matter how im- 
pressive a research prototype looks, we believe there is 
still a lot of research to be done in order to scale to 
real-world use. 

So: “Can we build a research prototype in a toy do- 
main and expect it to work on a real-world problem?” 
The answer is: “Probably not”. 

6 Conclusions 

KITSS is a domain-specific system to generate exe- 
cutable functional tests using the KBSE paradigm. Its 
main difference to previous approaches is that it covers 
a real-world domain instead of a toy domain. However, 
therefore the initial goal of automatic programming 
had to be limited to the easier problem of automatic 
testing. 

KITSS converts input tests into a formal represen- 
tation interactively with the user. To achieve this, 
we needed to augment the static domain model repre- 
sented in a terminological logic with a dynamic model 
written in a temporal logic. Knowledge acquisition has 
been performed manually and is a major problem that 
has not been addressed yet. 

In general, scaling from a research prototype to a 
real-world system involves much additional research 
before the actual technology transfer can begin. To 
achieve such scaling, we had to further move toward 
more user interaction. Although scaling-up remains a 
hard task, KITSS demonstrates that our KBSE ap- 
proach chosen for this complex application is feasible. 
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